Strokes localization by m-array decoding and fast image matching

ABSTRACT

Systems and methods that determine a path of a pen tip as the pen tip is moved across a document are described. The document is watermarked with a maze pattern from which encoded position information is determined. A sequence of images is captured by a camera that is located in a pen. The path of the pen tip is determined by decoding the associated maze pattern and by matching the captured images with document images. If the position coordinates of any frame that is associated with a stroke cannot be determined from m-array decoding, the frames are transformed and then matched with an area of a document image. Once the position coordinates of at least one frame is determined, the position coordinates of other frames are determined by matching the frames in a neighboring area.

TECHNICAL FIELD

The present invention relates to interacting with a medium using adigital pen. More particularly, the present invention relates todetermining the location of a digital pen during interaction with one ormore surfaces.

BACKGROUND

Computer users are accustomed to using a mouse and keyboard as a way ofinteracting with a personal computer. While personal computers provide anumber of advantages over written documents, most users continue toperform certain functions using printed paper. Some of these functionsinclude reading and annotating written documents. In the case ofannotations, the printed document assumes a greater significance becauseof the annotations placed on it by the user. One of the difficulties,however, with having a printed document with annotations is the laterneed to have the annotations entered back into the electronic form ofthe document. This requires the original user or another user to wadethrough the annotations and enter them into a personal computer. In somecases, a user will scan in the annotations and the original text,thereby creating a new document. These multiple steps make theinteraction between the printed document and the electronic version ofthe document difficult to handle on a repeated basis. Further,scanned-in images are frequently non-modifiable. There may be no way toseparate the annotations from the original text. This makes using theannotations difficult. Accordingly, an improved way of handlingannotations is needed.

One technique of capturing handwritten information is by using a penwhose location may be determined during writing. One pen that providesthis capability is the Anoto pen by Anoto Inc. This pen functions byusing a camera to capture an image of paper encoded with a predefinedpattern. An example of the image pattern is shown in FIG. 11. Thispattern is used by the Anoto pen (by Anoto Inc.) to determine a locationof a pen on a piece of paper. However, it is unclear how efficient thedetermination of the location is with the system used by the Anoto pen.To provide efficient determination of the location of the capturedimage, a system is needed that provides efficient decoding of thecaptured image.

When reviewing a document, a user may annotate the document by moving apen tip with respect to the document. The path of the pen tip maycomprise a plurality of strokes, where each stroke corresponds to aseries of captured images. Hence, there is a real need in the industryto identify the path of the pen in order to process the annotation on adocument.

SUMMARY

Aspects of the present invention provide system and methods thatdetermine a path of a pen tip as the pen tip is moved across a document.With an aspect of the invention, the document is watermarked with a mazepattern from which encoded position information is determined.

With another aspect of the invention, a sequence of images is capturedby a camera that is located in a pen. The path of the pen tip isdetermined by decoding the associated maze pattern (m-array) and bymatching the captured images with document images.

With another aspect of the invention, if the position coordinates of anyframe (corresponding to a captured image) that is associated with astroke cannot be determined from m-array decoding, the frames aretransformed and then matched with an area of a document image. Once theposition coordinates of at least one frame are determined, the positioncoordinates of other frames may be determined by matching the frames ina neighboring area.

With another aspect of the invention, the path of the pen tip(corresponding to a stroke) is mapped from a frame center using aperspective transform and calibration parameters. The perspectivetransform is obtained from maze pattern analysis and by matchingcamera-captured images with document images.

BRIEF DESCRIPTION OF DRAWINGS

The foregoing summary of the invention, as well as the followingdetailed description of preferred embodiments, is better understood whenread in conjunction with the accompanying drawings, which are includedby way of example, and not by way of limitation with regard to theclaimed invention.

FIG. 1 shows a general description of a computer that may be used inconjunction with embodiments of the present invention.

FIGS. 2A and 2B show an image capture system and corresponding capturedimage in accordance with embodiments of the present invention.

FIGS. 3A through 3F show various sequences and folding techniques inaccordance with embodiments of the present invention.

FIGS. 4A through 4E show various encoding systems in accordance withembodiments of the present invention.

FIGS. 5A through 5D show four possible resultant corners associated withthe encoding system according to FIGS. 4A and 4B.

FIG. 6 shows rotation of a captured image portion in accordance withembodiments of the present invention.

FIG. 7 shows various angles of rotation used in conjunction with thecoding system of FIGS. 4A through 4E.

FIG. 8 shows a process for determining the location of a captured arrayin accordance with embodiments of the present invention.

FIG. 9 shows a method for determining the location of a captured imagein accordance with embodiments of the present invention.

FIG. 10 shows another method for determining the location of a capturedimage in accordance with embodiments of the present invention.

FIG. 11 shows a representation of encoding space in a document accordingto prior art.

FIG. 12 shows a flow diagram for decoding extracted bits from a capturedimage in accordance with embodiments of the present invention.

FIG. 13 shows bit selection of extracted bits from a captured image inaccordance with embodiments of the present invention.

FIG. 14 shows an apparatus for decoding extracted bits from a capturedimage in accordance with embodiments of the present invention.

FIG. 15 shows a method for determining a path of a pen-tip from asequence of captured frames in accordance with an embodiment of thepresent invention.

FIG. 16 shows an apparatus for determining a path of a pen-tip from asequence of captured frames in accordance with an embodiment of thepresent invention.

FIG. 17 shows an example of a captured image in accordance with anembodiment of the invention.

FIG. 18 shows an example of a captured image comprising a text componentin accordance with an embodiment of the invention.

FIG. 19 shows a maze pattern analysis of a captured image in accordancewith an embodiment of the invention.

FIG. 20 shows a maze pattern analysis of a captured image comprising atext component in accordance with an embodiment of the invention.

FIG. 21 shows a result of analyzing a document image in accordance withan embodiment of the invention.

FIG. 22 shows a result of a global localization for an exemplary strokein accordance with an embodiment of the invention.

FIG. 23 shows a result of a local localization for an exemplary strokein accordance with an embodiment of the invention.

FIG. 24 shows a recovered stroke of a pen tip and a path ofcorresponding centers of captured images in accordance with anembodiment of the invention.

FIG. 25 shows an actual stroke that corresponds to the recovered strokeas shown in FIG. 24.

DETAILED DESCRIPTION

Aspects of the present invention relate to determining the location of acaptured image in relation to a larger image. The location determinationmethod and system described herein may be used in combination with amulti-function pen.

The following is separated by subheadings for the benefit of the reader.The subheadings include: terms, general-purpose computer, imagecapturing pen, encoding of array, decoding, error correction, locationdetermination, m-array decoding, and architecture for determining a pathof a pen.

Terms

Pen—any writing implement that may or may not include the ability tostore ink. In some examples, a stylus with no ink capability may be usedas a pen in accordance with embodiments of the present invention.

Camera—an image capture system that may capture an image from paper orany other medium.

General Purpose Computer

FIG. 1 is a functional block diagram of an example of a conventionalgeneral-purpose digital computing environment that can be used toimplement various aspects of the present invention. In FIG. 1, acomputer 100 includes a processing unit 110, a system memory 120, and asystem bus 130 that couples various system components including thesystem memory to the processing unit 110. The system bus 130 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory 120 includes read only memory (ROM)140 and random access memory (RAM) 150.

A basic input/output system 160 (BIOS), containing the basic routinesthat help to transfer information between elements within the computer100, such as during start-up, is stored in the ROM 140. The computer 100also includes a hard disk drive 170 for reading from and writing to ahard disk (not shown), a magnetic disk drive 180 for reading from orwriting to a removable magnetic disk 190, and an optical disk drive 191for reading from or writing to a removable optical disk 192 such as a CDROM or other optical media. The hard disk drive 170, magnetic disk drive180, and optical disk drive 191 are connected to the system bus 130 by ahard disk drive interface 192, a magnetic disk drive interface 193, andan optical disk drive interface 194, respectively. The drives and theirassociated computer-readable media provide nonvolatile storage ofcomputer readable instructions, data structures, program modules andother data for the personal computer 100. It will be appreciated bythose skilled in the art that other types of computer readable mediathat can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, Bernoullicartridges, random access memories (RAMs), read only memories (ROMs),and the like, may also be used in the example operating environment.

A number of program modules can be stored on the hard disk drive 170,magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including anoperating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a microphone, joystick, game pad, satellite dish, scanner orthe like. These and other input devices are often connected to theprocessing unit 110 through a serial port interface 106 that is coupledto the system bus, but may be connected by other interfaces, such as aparallel port, game port or a universal serial bus (USB). Further still,these devices may be coupled directly to the system bus 130 via anappropriate interface (not shown). A monitor 107 or other type ofdisplay device is also connected to the system bus 130 via an interface,such as a video adapter 108. In addition to the monitor, personalcomputers typically include other peripheral output devices (not shown),such as speakers and printers. In a preferred embodiment, a pendigitizer 165 and accompanying pen or stylus 166 are provided in orderto digitally capture freehand input. Although a direct connectionbetween the pen digitizer 165 and the serial port is shown, in practice,the pen digitizer 165 may be coupled to the processing unit 110directly, via a parallel port or other interface and the system bus 130as known in the art. Furthermore, although the digitizer 165 is shownapart from the monitor 107, it is preferred that the usable input areaof the digitizer 165 be co-extensive with the display area of themonitor 107. Further still, the digitizer 165 may be integrated in themonitor 107, or may exist as a separate device overlaying or otherwiseappended to the monitor 107.

The computer 100 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer109. The remote computer 109 can be a server, a router, a network PC, apeer device or other common network node, and typically includes many orall of the elements described above relative to the computer 100,although only a memory storage device 111 has been illustrated inFIG. 1. The logical connections depicted in FIG. 1 include a local areanetwork (LAN) 112 and a wide area network (WAN) 113. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

When used in a LAN networking environment, the computer 100 is connectedto the local network 112 through a network interface or adapter 114.When used in a WAN networking environment, the personal computer 100typically includes a modem 115 or other means for establishing acommunications over the wide area network 113, such as the Internet. Themodem 115, which may be internal or external, is connected to the systembus 130 via the serial port interface 106. In a networked environment,program modules depicted relative to the personal computer 100, orportions thereof, may be stored in the remote memory storage device.

It will be appreciated that the network connections shown areillustrative and other techniques for establishing a communications linkbetween the computers can be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP, Bluetooth,IEEE 802.11x and the like is presumed, and the system can be operated ina client-server configuration to permit a user to retrieve web pagesfrom a web-based server. Any of various conventional web browsers can beused to display and manipulate data on web pages.

Image Capturing Pen

Aspects of the present invention include placing an encoded data streamin a displayed form that represents the encoded data stream. (Forexample, as will be discussed with FIG. 4B, the encoded data stream isused to create a graphical pattern.) The displayed form may be printedpaper (or other physical medium) or may be a display projecting theencoded data stream in conjunction with another image or set of images.For example, the encoded data stream may be represented as a physicalgraphical image on the paper or a graphical image overlying thedisplayed image (e.g., representing the text of a document) or may be aphysical (non-modifiable) graphical image on a display screen (so anyimage portion captured by a pen is locatable on the display screen).

This determination of the location of a captured image may be used todetermine the location of a user's interaction with the paper, medium,or display screen. In some aspects of the present invention, the pen maybe an ink pen writing on paper. In other aspects, the pen may be astylus with the user writing on the surface of a computer display. Anyinteraction may be provided back to the system with knowledge of theencoded image on the document or supporting the document displayed onthe computer screen. By repeatedly capturing images with a camera in thepen or stylus as the pen or stylus traverses a document, the system cantrack movement of the stylus being controlled by the user. The displayedor printed image may be a watermark associated with the blank orcontent-rich paper or may be a watermark associated with a displayedimage or a fixed coding overlying a screen or built into a screen.

FIGS. 2A and 2B show an illustrative example of pen 201 with a camera203. Pen 201 includes a tip 202 that may or may not include an inkreservoir. Camera 203 captures an image 204 from surface 207. Pen 201may further include additional sensors and/or processors as representedin broken box 206. These sensors and/or processors 206 may also includethe ability to transmit information to another pen 201 and/or a personalcomputer (for example, via Bluetooth or other wireless protocols).

FIG. 2B represents an image as viewed by camera 203. In one illustrativeexample, the field of view of camera 203 (i.e., the resolution of theimage sensor of the camera) is 32×32 pixels (where N=32). In theembodiment, a captured image (32 pixels by 32 pixels) corresponds to anarea of approximately 5 mm by 5 mm of the surface plane captured bycamera 203. Accordingly, FIG. 2B shows a field of view of 32 pixels longby 32 pixels wide. The size of N is adjustable, such that a larger Ncorresponds to a higher image resolution. Also, while the field of viewof the camera 203 is shown as a square for illustrative purposes here,the field of view may include other shapes as is known in the art.

The images captured by camera 203 may be defined as a sequence of imageframes {I_(i)}, where I_(i) is captured by the pen 201 at sampling timet_(i). The sampling rate may be large or small, depending on systemconfiguration and performance requirement. The size of the capturedimage frame may be large or small, depending on system configuration andperformance requirement.

The image captured by camera 203 may be used directly by the processingsystem or may undergo pre-filtering. This pre-filtering may occur in pen201 or may occur outside of pen 201 (for example, in a personalcomputer).

The image size of FIG. 2B is 32×32 pixels. If each encoding unit size is3×3 pixels, then the number of captured encoded units would beapproximately 100 units. If the encoding unit size is 5×5 pixels, thenthe number of captured encoded units is approximately 36.

FIG. 2A also shows the image plane 209 on which an image 210 of thepattern from location 204 is formed. Light received from the pattern onthe object plane 207 is focused by lens 208. Lens 208 may be a singlelens or a multi-part lens system, but is represented here as a singlelens for simplicity. Image capturing sensor 211 captures the image 210.

The image sensor 211 may be large enough to capture the image 210.Alternatively, the image sensor 211 may be large enough to capture animage of the pen tip 202 at location 212. For reference, the image atlocation 212 is referred to as the virtual pen tip. It is noted that thevirtual pen tip location with respect to image sensor 211 is fixedbecause of the constant relationship between the pen tip, the lens 208,and the image sensor 211.

The following transformation F_(S→P) transforms position coordinates inthe image captured by camera to position coordinates in the real imageon the paper:L _(paper) =F _(S→P)(L _(Sensor))

During writing, the pen tip and the paper are on the same plane.Accordingly, the transformation from the virtual pen tip to the real pentip is also F_(S→P):L _(pentip) =F _(S→P)(L _(virtual-pentip))

The transformation F_(S→P) may be estimated as an affine transform. Thissimplifies as:

$F_{s->P}^{\prime} = \left\{ \begin{matrix}{\frac{s_{x}\sin\;\theta_{y}}{{\cos\;\theta_{x}\sin\;\theta_{y}} - {\cos\;\theta_{y}\sin\;\theta_{x}}},} & {{- \frac{s_{x}\cos\;\theta_{y}}{{\cos\;\theta_{x}\sin\;\theta_{y}} - {\cos\;\theta_{y}\sin\;\theta_{x}}}},} & 0 \\{{- \frac{s_{y}\sin\;\theta_{x}}{{\cos\;\theta_{x}\sin\;\theta_{y}} - {\cos\;\theta_{y}\sin\;\theta_{x}}}},} & {\frac{s_{y}\sin\;\theta_{x}}{{\cos\;\theta_{x}\sin\;\theta_{y}} - {\cos\;\theta_{y}\sin\;\theta_{x}}},} & 0 \\{0,} & {0,} & 1\end{matrix} \right)$as the estimation of F_(S→P), in which θ_(x), θ_(y), s_(x), and s_(y)are the rotation and scale of two orientations of the pattern capturedat location 204. Further, one can refine F′_(S→P) by matching thecaptured image with the corresponding real image on paper. “Refine”means to get a more precise estimation of the transformation F_(S→P) bya type of optimization algorithm referred to as a recursive method. Therecursive method treats the matrix F′_(S→P) as the initial value. Therefined estimation describes the transformation between S and P moreprecisely.

Next, one can determine the location of virtual pen tip by calibration.

One places the pen tip 202 on a fixed location L_(pentip) on paper.Next, one tilts the pen, allowing the camera 203 to capture a series ofimages with different pen poses. For each image captured, one may obtainthe transformation F_(S→P). From this transformation, one can obtain thelocation of the virtual pen tip L_(virtual-pentip):L _(virtual-pentip) =F _(P→S)(L _(pentip))where L_(pentip) is initialized as (0, 0) andF _(P→S)=(F _(S→P))⁻¹

By averaging the L_(virtual-pentip) obtained from each image, a locationof the virtual pen tip L_(virtual-pentip) may be determined. WithL_(virtual-pentip), one can get a more accurate estimation ofL_(pentip). After several times of iteration, an accurate location ofvirtual pen tip L_(virtual-pentip) may be determined.

The location of the virtual pen tip L_(virtual-pentip) is now known. Onecan also obtain the transformation F_(S→P) from the images captured.Finally, one can use this information to determine the location of thereal pen tip L_(pentip):L _(pentip) =F _(S→P)(L _(virtual-pentip))Encoding of Array

A two-dimensional array may be constructed by folding a one-dimensionalsequence. Any portion of the two-dimensional array containing a largeenough number of bits may be used to determine its location in thecomplete two-dimensional array. However, it may be necessary todetermine the location from a captured image or a few captured images.So as to minimize the possibility of a captured image portion beingassociated with two or more locations in the two-dimensional array, anon-repeating sequence may be used to create the array. One property ofa created sequence is that the sequence does not repeat over a length(or window) n. The following describes the creation of theone-dimensional sequence then the folding of the sequence into an array.

Sequence Construction

A sequence of numbers may be used as the starting point of the encodingsystem. For example, a sequence (also referred to as an m-sequence) maybe represented as a q-element set in field F_(q). Here, q=p^(n) wheren≧1 and p is a prime number. The sequence or m-sequence may be generatedby a variety of different techniques including, but not limited to,polynomial division. Using polynomial division, the sequence may bedefined as follows:

$\frac{R_{l}(x)}{P_{n}(x)}$where P_(n)(x) is a primitive polynomial of degree n in field F_(q)[x](having q^(n) elements). R_(l)(x) is a nonzero polynomial of degree l(where l<n) in field F_(q)[x]. The sequence may be created using aniterative procedure with two steps: first, dividing the two polynomials(resulting in an element of field F_(q)) and, second, multiplying theremainder by x. The computation stops when the output begins to repeat.This process may be implemented using a linear feedback shift registeras set forth in an article by Douglas W. Clark and Lih-Jyh Weng,“Maximal and Near-Maximal Shift Register Sequences: Efficient EventCounters and Easy Discrete Logarithms,” IEEE Transactions on Computers43.5 (May 1994, pp 560-568). In this environment, a relationship isestablished between cyclical shifting of the sequence and polynomialR_(l)(x): changing R_(l)(x) only cyclically shifts the sequence andevery cyclical shifting corresponds to a polynomial R_(l)(x). One of theproperties of the resulting sequence is that, the sequence has a periodof q^(n)−1 and within a period, over a width (or length) n, any portionexists once and only once in the sequence. This is called the “windowproperty”. Period q^(n)−1 is also referred to as the length of thesequence and n as the order of the sequence.

The process described above is but one of a variety of processes thatmay be used to create a sequence with the window property.

Array Construction

The array (or m-array) that may be used to create the image (of which aportion may be captured by the camera) is an extension of theone-dimensional sequence or m-sequence. Let A be an array of period (m₁,m₂), namely A(k+m₁, l)=A(k,l+m₂)=A(k,l). When an n₁×n₂ window shiftsthrough a period of A, all the nonzero n₁×n₂ matrices over F_(q) appearonce and only once. This property is also referred to as a “windowproperty” in that each window is unique.

A binary array (or m-array) may be constructed by folding the sequence.One approach is to obtain a sequence then fold it to a size of m₁×m₂where the length of the array is L=m₁×m₂=2^(n)−1. Alternatively, one maystart with a predetermined size of the space that one wants to cover(for example, one sheet of paper, 30 sheets of paper or the size of acomputer monitor), determine the area (m₁×m₂), then use the size to letL≧m₁×m₂, where L=2^(n)−1.

A variety of different folding techniques may be used. For example,FIGS. 3A through 3C show three different sequences. Each of these may befolded into the array shown as FIG. 3D. The three different foldingmethods are shown as the overlay in FIG. 3D and as the raster paths inFIGS. 3E and 3F. We adopt the folding method shown in FIG. 3D.

To create the folding method as shown in FIG. 3D, one creates a sequence{a_(i)} of length L and order n. Next, an array {b_(kl)} of size m₁×m₂,where gcd(m₁, m₂)=1 and L=m₁×m₂, is created from the sequence {a_(i)} byletting each bit of the array be calculated as shown by equation 1:b_(kl)=a_(i), where k=i mod(m₁), l=i mod(m₂), i=0, . . . , L−1.  (1)

This folding approach may be alternatively expressed as laying thesequence on the diagonal of the array, then continuing from the oppositeedge when an edge is reached.

FIG. 4A shows sample encoding techniques that may be used to encode thearray of FIG. 3D. It is appreciated that other encoding techniques maybe used. For example, an alternative coding technique is shown in FIG.11.

Referring to FIG. 4A, a first bit 401 (for example, “1”) is representedby a column of dark ink. A second bit 402 (for example, “0”) isrepresented by a row of dark ink. It is appreciated that any color inkmay be used to represent the various bits. The only requirement in thecolor of the ink chosen is that it provides a significant contrast withthe background of the medium to be differentiable by an image capturesystem. The bits in FIG. 4A are represented by a 3×3 matrix of cells.The size of the matrix may be modified to be any size as based on thesize and resolution of an image capture system. Alternativerepresentation of bits 0 and 1 are shown in FIGS. 4C-4E. It isappreciated that the representation of a one or a zero for the sampleencodings of FIGS. 4A-4E may be switched without effect. FIG. 4C showsbit representations occupying two rows or columns in an interleavedarrangement. FIG. 4D shows an alternative arrangement of the pixels inrows and columns in a dashed form. Finally FIG. 4E shows pixelrepresentations in columns and rows in an irregular spacing format(e.g., two dark dots followed by a blank dot).

Referring back to FIG. 4A, if a bit is represented by a 3×3 matrix andan imaging system detects a dark row and two white rows in the 3×3region, then a zero is detected (or one). If an image is detected with adark column and two white columns, then a one is detected (or a zero).

Here, more than one pixel or dot is used to represent a bit. Using asingle pixel (or bit) to represent a bit is fragile. Dust, creases inpaper, non-planar surfaces, and the like create difficulties in readingsingle bit representations of data units. However, it is appreciatedthat different approaches may be used to graphically represent the arrayon a surface. Some approaches are shown in FIGS. 4C through 4E. It isappreciated that other approaches may be used as well. One approach isset forth in FIG. 11 using only space-shifted dots.

A bit stream is used to create the graphical pattern 403 of FIG. 4B.Graphical pattern 403 includes 12 rows and 18 columns. The rows andcolumns are formed by a bit stream that is converted into a graphicalrepresentation using bit representations 401 and 402. FIG. 4B may beviewed as having the following bit representation:

$\quad\begin{bmatrix}0 & 1 & 0 & 1 & 0 & 1 & 1 & 1 & 0 \\1 & 1 & 0 & 1 & 1 & 0 & 0 & 1 & 0 \\0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 1 \\1 & 0 & 1 & 1 & 0 & 1 & 1 & 0 & 0\end{bmatrix}$Decoding

When a person writes with the pen of FIG. 2A or moves the pen close tothe encoded pattern, the camera captures an image. For example, pen 201may utilize a pressure sensor as pen 201 is pressed against paper andpen 201 traverses a document on the paper. The image is then processedto determine the orientation of the captured image with respect to thecomplete representation of the encoded image and extract the bits thatmake up the captured image.

For the determination of the orientation of the captured image relativeto the whole encoded area, one may notice that not all the fourconceivable corners shown in FIG. 5A-5D can present in the graphicalpattern 403. In fact, with the correct orientation, the type of cornershown in FIG. 5A cannot exist in the graphical pattern 403. Therefore,the orientation in which the type of corner shown in FIG. 5A is missingis the right orientation.

Continuing to FIG. 6, the image captured by a camera 601 may be analyzedand its orientation determined so as to be interpretable as to theposition actually represented by the image 601. First, image 601 isreviewed to determine the angle θ needed to rotate the image so that thepixels are horizontally and vertically aligned. It is noted thatalternative grid alignments are possible including a rotation of theunderlying grid to a non-horizontal and vertical arrangement (forexample, 45 degrees). Using a non-horizontal and vertical arrangementmay provide the probable benefit of eliminating visual distractions fromthe user, as users may tend to notice horizontal and vertical patternsbefore others. For purposes of simplicity, the orientation of the grid(horizontal and vertical and any other rotation of the underlying grid)is referred to collectively as the predefined grid orientation.

Next, image 601 is analyzed to determine which corner is missing. Therotation amount o needed to rotate image 601 to an image ready fordecoding 603 is shown as o=(θ plus a rotation amount {defined by whichcorner missing}). The rotation amount is shown by the equation in FIG.7. Referring back to FIG. 6, angle θ is first determined by the layoutof the pixels to arrive at a horizontal and vertical (or otherpredefined grid orientation) arrangement of the pixels and the image isrotated as shown in 602. An analysis is then conducted to determine themissing corner and the image 602 rotated to the image 603 to set up theimage for decoding. Here, the image is rotated 90 degreescounterclockwise so that image 603 has the correct orientation and canbe used for decoding.

It is appreciated that the rotation angle θ may be applied before orafter rotation of the image 601 to account for the missing corner. It isalso appreciated that by considering noise in the captured image, allfour types of corners may be present. We may count the number of cornersof each type and choose the type that has the least number as the cornertype that is missing.

Finally, the code in image 603 is read out and correlated with theoriginal bit stream used to create image 403. The correlation may beperformed in a number of ways. For example, it may be performed by arecursive approach in which a recovered bit stream is compared againstall other bit stream fragments within the original bit stream. Second, astatistical analysis may be performed between the recovered bit streamand the original bit stream, for example, by using a Hamming distancebetween the two bit streams. It is appreciated that a variety ofapproaches may be used to determine the location of the recovered bitstream within the original bit stream.

Once one has the recovered bits, one needs to locate the captured imagewithin the original array (for example, the one shown in FIG. 4B). Theprocess of determining the location of a segment of bits within theentire array is complicated by a number of items. First, the actual bitsto be captured may be obscured (for example, the camera may capture animage with document content that obscures the code). Second, dust,creases, reflections, and the like may also create errors in thecaptured image. These errors make the localization process moredifficult. In this regard, the image capture system may need to functionwith non-sequential bits extracted from the image. The followingrepresents a method for operating with non-sequential bits from theimage.

Let the sequence (or m-sequence) I correspond to the power seriesI(x)=1/P_(n)(x), where n is the order of the m-sequence, and thecaptured image contains K bits of I b=(b₀ b₁ b₂ . . . b_(K−1))^(t),where K≧n and the superscript t represents a transpose of the matrix orvector. The location s of the K bits is just the number of cyclic shiftsof I so that b₀ is shifted to the beginning of the sequence. Then thisshifted sequence R corresponds to the power series x^(s)/P_(n)(x), orR=T^(s)(I), where T is the cyclic shift operator. We find this sindirectly. The polynomials modulo P_(n)(x) form a field. It isguaranteed that x^(s)≡r₀+r₁x+ . . . r_(n−1)x^(n−1)mod(P_(n)(x)).Therefore, we may find (r₀, r₁, . . . , r_(n−1)) and then solve for s.

The relationship x^(s)≡r₀+r₁x+ . . . r_(n−1)x^(n−1)mod(P_(n)(x)) impliesthat R=r₀+r₁T(I)+ . . . +r_(n−1)T^(n−1)(I). Written in a binary linearequation, it becomes:R=r^(t)A  (2)where r=(r₀ r₁ r₂ . . . r_(n−1))^(t), and A=(I T(I). . . T^(n−1)(I))^(t)which consists of the cyclic shifts of I from 0-shift to (n−1)-shift.Now only sparse K bits are available in R to solve r. Let the indexdifferences between b_(i) and b₀ in R be k_(i), i=1,2, . . . , k−1, thenthe 1^(st) and (k_(i)+1)-th elements of R, i=1,2, . . . , k−1, areexactly b₀, b₁, . . . , b_(k−1). By selecting the 1^(st) and(k_(i)+1)-th columns of A, i=1,2, . . . , k−1, the following binarylinear equation is formed:b^(t)=r^(t)M  (3)

-   -   where M is an n×K sub-matrix of A.

If b is error-free, the solution of r may be expressed as:r^(t)={tilde over (b)}^(t){tilde over (M)}⁻¹  (4)where {tilde over (M)} is any non-degenerate n×n sub-matrix of M and{tilde over (b)} is the corresponding sub-vector of b.

With known r, we may use the Pohlig-Hellman-Silver algorithm as noted byDouglas W. Clark and Lih-Jyh Weng, “Maximal and Near-Maximal ShiftRegister Sequences: Efficient Event Counters and Easy DiscreteLogarithms,” IEEE Transactions on Computers 43.5 (May 1994, pp 560-568)to find s so that x^(s)≡r₀+r₁x+ . . . r_(n−1)x^(n−1)mod(P_(n)(x)).

As matrix A (with the size of n by L, where L=2^(n)−1) may be huge, weshould avoid storing the entire matrix A. In fact, as we have seen inthe above process, given extracted bits with index difference k_(i),only the first and (k_(i)+1)-th columns of A are relevant to thecomputation. Such choices of k_(i) is quite limited, given the size ofthe captured image. Thus, only those columns that may be involved incomputation need to be saved. The total number of such columns is muchsmaller than L (where L=2^(n)−1 is the length of the m-sequence).

Error Correction

If errors exist in b, then the solution of r becomes more complex.Traditional methods of decoding with error correction may not readilyapply, because the matrix M associated with the captured bits may changefrom one captured image to another.

We adopt a stochastic approach. Assuming that the number of error bitsin b, n_(e), is relatively small compared to K, then the probability ofchoosing the correct n bits from the K bits of b and the correspondingsub-matrix {tilde over (M)} of M being non-degenerate is high.

When the n bits chosen are all correct, the Hamming distance betweenb^(t) and r^(t)M, or the number of error bits associated with r, shouldbe minimal, where r is computed via equation (4). Repeating the processfor several times, it is likely that the correct r that results in theminimal error bits can be identified.

If there is only one r that is associated with the minimum number oferror bits, then it is regarded as the correct solution. Otherwise, ifthere is more than one r that is associated with the minimum number oferror bits, the probability that n_(e) exceeds the error correctingability of the code generated by M is high and the decoding processfails. The system then may move on to process the next captured image.In another implementation, information about previous locations of thepen can be taken into consideration. That is, for each captured image, adestination area where the pen may be expected next can be identified.For example, if the user has not lifted the pen between two imagecaptures by the camera, the location of the pen as determined by thesecond image capture should not be too far away from the first location.Each r that is associated with the minimum number of error bits can thenbe checked to see if the location s computed from r satisfies the localconstraint, i.e., whether the location is within the destination areaspecified.

If the location s satisfies the local constraint, the X, Y positions ofthe extracted bits in the array are returned. If not, the decodingprocess fails.

FIG. 8 depicts a process that may be used to determine a location in asequence (or m-sequence) of a captured image. First, in step 801, a datastream relating to a captured image is received. In step 802,corresponding columns are extracted from A and a matrix M isconstructed.

In step 803, n independent column vectors are randomly selected from thematrix M and vector r is determined by solving equation (4). Thisprocess is performed Q times (for example, 100 times) in step 804. Thedetermination of the number of loop times is discussed in the sectionLoop Times Calculation later.

In step 805, r is sorted according to its associated number of errorbits. The sorting can be done using a variety of sorting algorithms asknown in the art. For example, a selection sorting algorithm may beused. The selection sorting algorithm is beneficial when the number Q isnot large. However, if Q becomes large, other sorting algorithms (forexample, a merge sort) that handle larger numbers of items moreefficiently may be used.

The system then determines in step 806 whether error correction wasperformed successfully, by checking whether multiple r's are associatedwith the minimum number of error bits. If yes, an error is returned instep 809, indicating the decoding process failed. If not, the position sof the extracted bits in the sequence (or m-sequence) is calculated instep 807, for example, by using the Pohig-Hellman-Silver algorithm.

Next, the (X,Y) position in the array is calculated as: x=s mod m₁ andy=s mod m₂ and the results are returned in step 808.

Location Determination

FIG. 9 shows a process for determining the location of a pen tip. Theinput is an image captured by a camera and the output may be a positioncoordinates of the pen tip. Also, the output may include (or not) otherinformation such as a rotation angle of the captured image.

In step 901, an image is received from a camera. Next, the receivedimage may be optionally preprocessed in step 902 (as shown by the brokenoutline of step 902) to adjust the contrast between the light and darkpixels and the like.

Next, in step 903, the image is analyzed to determine the bit streamwithin it.

Next, in step 904, n bits are randomly selected from the bit stream formultiple times and the location of the received bit stream within theoriginal sequence (or m-sequence) is determined.

Finally, once the location of the captured image is determined in step904, the location of the pen tip may be determined in step 905.

FIG. 10 gives more details about 903 and 904 and shows the approach toextract the bit stream within a captured image. First, an image isreceived from the camera in step 1001. The image then may optionallyundergo image preprocessing in step 1002 (as shown by the broken outlineof step 1002). The pattern is extracted in step 1003. Here, pixels onthe various lines may be identified and the orientation of the pattern(corresponding to the angle θ) passing through the pixel can beestimated.

Next, the received image is analyzed in step 1004 to determine theunderlying grid lines. If grid lines are found in step 1005, then thecode is extracted from the pattern in step 1006. The code is thendecoded in step 1007 and the location of the pen tip is determined instep 1008. If no grid lines were found in step 1005, then an error isreturned in step 1009.

Outline of Enhanced Decoding and Error Correction Algorithm

With an embodiment of the invention as shown in FIG. 12, given extractedbits 1201 from a captured image (corresponding to a captured array) andthe destination area, a variation of an m-array decoding and errorcorrection process decodes the X,Y position. FIG. 12 shows a flowdiagram of process 1200 of this enhanced approach. Process 1200comprises two components 1251 and 1253.

-   Decode Once. Component 1251 includes three parts.    -   random bit selection: randomly selects a subset of the extracted        bits 1201 (step 1203)    -   decode the subset (step 1205)    -   determine X, Y position with local constraint (step 1209)-   Decoding with Smart Bit Selection. Component 1253 includes four    parts.    -   smart bit selection: selects another subset of the extracted        bits (step 1217)    -   decode the subset (step 1219)    -   adjust the number of iterations (loop times) of step 1217 and        step 1219 (step 1221)    -   determine X,Y position with local constraint (step 1225)

The embodiment of the invention utilizes a discreet strategy to selectbits, adjusts the number of loop iterations, and determines the X,Yposition (location coordinates) in accordance with a local constraint,which is provided to process 1200. With both components 1251 and 1253,steps 1205 and 1219 (“Decode Once”) utilize equation (4) to compute r.

Let {circumflex over (b)} be decoded bits, that is:{circumflex over (b)}^(t)=r^(t)M  (5)

The difference between b and {circumflex over (b)} are the error bitsassociated with r.

FIG. 12 shows a flow diagram of process 1200 for decoding extracted bits1201 from a captured image in accordance with embodiments of the presentinvention. Process 1200 comprises components 1251 and 1253. Component1251 obtains extracted bits 1201 (comprising K bits) associated with acaptured image (corresponding to a captured array). In step 1203, n bits(where n is the order of the m-array) are randomly selected fromextracted bits 1201. In step 1205, process 1200 decodes once andcalculates r. In step 1207, process 1200 determines if error bits aredetected for b. If step 1207 determines that there are no error bits,X,Y coordinates of the position of the captured array are determined instep 1209. With step 1211, if the X,Y coordinates satisfy the localconstraint, i.e., coordinates that are within the destination area,process 1200 provides the X,Y position (such as to another process oruser interface) in step 1213. Otherwise, step 1215 provides a failureindication.

If step 1207 detects error bits in b, component 1253 is executed inorder to decode with error bits. Step 1217 selects another set of n bits(which differ by at least one bit from the n bits selected in step 1203)from extracted bits 1201. Steps 1221 and 1223 determine the number ofiterations (loop times) that are necessary for decoding the extractedbits. Step 1225 determines the position of the captured array by testingwhich candidates obtained in step 1219 satisfy the local constraint.Steps 1217-1225 will be discussed in more details.

Smart Bit Selection

Step 1203 randomly selects n bits from extracted bits 1201 (having Kbits), and solves for r₁. Using equation (5), decoded bits can becalculated. Let I₁={kε{1,2, . . . , K}|b_(k)={circumflex over (b)}_(k)},Ī₁={kε{1,2, . . . , K}|b_(k)≠{circumflex over (b)}_(k)}, where{circumflex over (b)}_(k) is the k^(th) bit of {circumflex over (b)},B₁={b_(k)|kεI₁} and B ₁={b_(k)|kεĪ₁}, that is, B₁ are bits that thedecoded results are the same as the original bits, and B ₁ are bits thatthe decoded results are different from the original bits, I₁ and Ī₁ arethe corresponding indices of these bits. It is appreciated that the samer₁ will be obtained when any n bits are selected from B₁. Therefore, ifthe next n bits are not carefully chosen, it is possible that theselected bits are a subset of B₁, thus resulting in the same r₁ beingobtained.

In order to avoid such a situation, step 1217 selects the next n bitsaccording to the following procedure:

-   -   1. Choose at least one bit from B ₁ 1303 and the rest of the        bits randomly from B₁ 1301 and B ₁ 1303, as shown in FIG. 13        corresponding to bit arrangement 1351. Process 1200 then solves        r₂ and finds B₂ 1305, 1309 and B ₂ 1307, 1311 by computing        {circumflex over (b)}₂ ^(t)=r₂ ^(t)M₂.    -   2. Repeat step 1. When selecting the next n bits, for every B        _(i) (i=1, 2, 3 . . . , x−1, where x is the current loop        number), there is at least one bit selected from B _(i). The        iteration terminates when no such subset of bits can be selected        or when the loop times are reached.        Loop Times Calculation

With the error correction component 1253, the number of requirediterations (loop times) is adjusted after each loop. The loop times isdetermined by the expected error rate. The expected error rate p_(e) inwhich not all the selected n bits are correct is:

$\begin{matrix}{p_{e} = {\left( {1 - \frac{C_{K - n_{e}}^{n}}{C_{K}^{n}}} \right)^{l\; t} \approx {- {\mathbb{e}}^{{- l}\;{t{(\frac{K - n}{K})}}^{n_{e}}}}}} & (6)\end{matrix}$where lt represents the loop times and is initialized by a constant, Kis the number of extracted bits from the captured array, n_(e)represents the minimum number of error bits incurred during theiteration of process 1200, n is the order of the m-array, and C_(K) ^(n)is the number of combinations in which n bits are selected from K bits.

In the embodiment, we want p_(e) to be less than e⁻⁵=0.0067. Incombination with (6), we have:

$\begin{matrix}{{l\; t_{i}} = {\min\left( {{l\; t_{i - 1}},{\frac{5}{\left( \frac{K - n}{K} \right)^{n_{e}}} + 1}} \right)}} & (7)\end{matrix}$Adjusting the loop times may significantly reduce the number ofiterations of process 1253 that are required for error correction.Determine X, Y Position with Local Constraint

In steps 1209 and 1225, the decoded position should be within thedestination area. The destination area is an input to the algorithm, andit may be of various sizes and places or simply the whole m-arraydepending on different applications. Usually it can be predicted by theapplication. For example, if the previous position is determined,considering the writing speed, the destination area of the current pentip should be close to the previous position. However, if the pen islifted, then its next position can be anywhere. Therefore, in this case,the destination area should be the whole m-array. The correct X,Yposition is determined by the following steps.

In step 1224 process 1200 selects r_(i) whose corresponding number oferror bits is less than:

$\begin{matrix}{N_{e} = \frac{\log_{10}\left( \frac{3}{l\; t} \right)}{{\log_{10}\left( \frac{K - n}{K} \right)} \times {\log_{10}\left( \frac{10}{l\; r} \right)}}} & (8)\end{matrix}$where lt is the actual loop times and lr represents the Local ConstraintRate calculated by:

$\begin{matrix}{{l\; r} = \frac{{area}\mspace{14mu}{of}\mspace{14mu}{destination}\mspace{14mu}{area}}{L}} & (9)\end{matrix}$where L is the length of the m-array.

Step 1224 sorts r_(i) in ascending order of the number of error bits.Steps 1225, 1211 and 1212 then finds the first r_(i) in which thecorresponding X,Y position is within the destination area. Steps 1225,1211 and 1212 finally returns the X,Y position as the result (throughstep 1213), or an indication that the decoding procedure failed (throughstep 1215).

Architecture for Determining Path of a Stroke (Strokes Localization)

FIG. 15 shows a method for determining a path of a pen-tip from asequence of captured frames 1551 in accordance with an embodiment of thepresent invention. In step 1501, a frame is processed so that documentcontents such as text are separated from other areas that contain onlymaze pattern cells. Also, the image (gray-scale) is normalized tocompensate for non-uniform illumination. In step 1503, m-array bits areextracted from visible maze pattern bars in a frame (captured image). Instep 1505, if the number of extracted bits (as determined in step 1503)is greater than the order of the embedded m-array, unique positioncoordinates (corresponding to the x-y position of the center of theframe) may be obtained by m-array decoding.

Step 1507 analyzes digital document 1553 in order to determine the areasof document 1553 in which the maze pattern is occluded by contents ofdocument 1553. (With the embodiment, digital document 1553 does notinclude annotations generated by a user.) If maze pattern cells areoccluded by the contents of document 1553, step 1505 may not be able toextract enough m-array bits to determine the x-y position of a frame. Byanalyzing the whole watermarked document 1553, process 1500 canascertain the areas of document 1553 where the x-y position cannot beuniquely determined. This analysis may be used in step 1509 as will bediscussed.

In step 1511, if the x-y position of any frame that is associated with astroke cannot be determined from m-array decoding (corresponding to step1505), process 1500 may not be able to locate the position of the frame.In such cases, step 1509 performs global localization in which capturedimages can be warped by an affine transform obtained by maze patternanalysis (step 1503) and then can be matched with an area of documentimage 1553 where a substantial amount of maze pattern cells are occludedby the content of the document. Once one frame is successfully matched,local localization (step 1517 as will be discussed) is utilized tolocate the entire stroke (corresponding to a series of frames).

In step 1517, frames whose x-y positions are not decoded in step 1505 ordetermined by global localization in step 1509 are further processed.The location of such frames should be near the location of frames whosex-y positions are determined by m-array decoding or global localization.The location of such frames is determined by matching the frames withthe neighboring area of already determined x-y positions. Theperspective transform F_(S→P) (as previously discussed) between capturedimages and document images is also obtained.

X-y positions obtained from m-array decoding (corresponding to step1505) and global/local localization (steps 1509 and 1517, respectively)represent the locations (x-y positions) of the centers of capturedimages. (An embodiment of the invention may support one, two or threedimensions. In the embodiment, two dimensions are supported so that aposition location corresponds to an x-y position.) In order to obtainthe x-y positions of the pen tip, a relationship between the pen tip(e.g., pen tip 202 as shown in FIG. 2) and the associated camera (e.g.,camera 203 as shown in FIG. 2) may need calibration. Step 1513 supportscalibration as will be discussed.

In step 1515 the x-y positions of the pen tip are determined by mappingthe x-y positions of image centers using the perspective transformobtained from local localization and calibration parameters. (An exampleof determining a path 1555 of a pen tip is shown in FIG. 26 as will bediscussed.)

FIG. 16 shows an apparatus 1600 for determining a path of a pen tip froma sequence of captured images (frames) in accordance with an embodimentof the present invention. In an embodiment of the invention, imagesegmentation 1601 performs step 1501, maze pattern analyzer 1603performs step 1503, m-array decoding module 1605 performs step 1505,matching module 1609 performs steps 1511, 1509, and 1517, watermarkeddocument analyzer 1607 performs step 1507, calibration module 1611performs step 1513, and mapping module 1613 performs step 1515.

Document image analysis and calibration may be performed off-line, orseparately before on-line image-capturing and processing. The othercomponents (e.g., image segmentation module 1601, maze pattern analyzer1603, m-array decoding module 1605, matching module 1609, and mappingmodule 1613) may be performed on-line or off-line. The functionality ofthe components is described in detail as follows.

Image Segmentation

Captured images may contain contents of a document such as text ordrawings, which are initially assessed in preprocessing. FIG. 17 showsan example of a captured image 1700 in accordance with an embodiment ofthe invention. FIG. 18 shows an example of a captured image comprising atext component (that includes text characters 1801 and 1803) inaccordance with an embodiment of the invention. If an image contains atext/drawing area, the text/drawing area is separated from the otherarea that contains only maze patterns or blank pixels.

In addition, the captured images (e.g., captured images 1700 and 1800)are normalized to compensate for the non-uniform illumination. FIG. 19shows a processed image 1900 corresponding to a normalization ofillumination of captured image 1700 (as shown in FIG. 17) in accordancewith an embodiment of the invention. FIG. 20 shows a processed image2000 corresponding to a normalization of illumination of captured image1800 in accordance with an embodiment of the invention.

Maze Pattern Analysis

The task of maze pattern analysis is to extract m-array bits from thevisible maze pattern bars (e.g., bars 401 and 402 as shown in FIG. 4A)in a captured image. FIG. 19 shows a maze pattern analysis 1900 ofcaptured image 1700 (as shown in FIG. 17) in accordance with anembodiment of the invention. Bit 1901 (corresponding to a bit value 1902of “0”) and bit 1903 (corresponding to a bit value of 1904 of “1”) aretwo members of the m-array bits. The m-array bits are organized in themaze pattern according to maze pattern grid lines, e.g., grid lines 1905and 1907. FIG. 20 shows a maze pattern analysis 2000 of captured image1800 (as shown in FIG. 18) in accordance with an embodiment of theinvention. (Note that m-array bits of some of the maze pattern cells inthe vicinity of text characters 1801 and 1803 may not be determined inthe example.)

FIGS. 19 and 20 show an illustration of maze pattern cells and mazepattern bars. The parameters of maze pattern grid lines (scale androtation along each dimension, i.e. affine transform) are firstcalculated, and then the original direction (or quadrant) in which themaze pattern is embedded is determined. Consequently, the m-array bitsinformation is determined based on the grid lines and bar directions.

m-Array Decoding

If the number of extracted bits obtained in step 1505 (as shown in FIG.15) is greater than the order of embedded m-array, a unique x-y positionmay be obtained by m-array decoding.

Watermarked Document Image Analysis

Maze patterns are possibly occluded by content of the document, whichmeans there may not be enough m-array bits that can be extracted fordecoding from a captured image. By analyzing the whole watermarkeddocument image, process 1500 or apparatus 1600 can determine in whicharea where the x-y position can be uniquely determined and in which areawhere x-y position cannot be uniquely determined. The analysis result isused in the global localization. FIG. 21 shows a result 2100 ofanalyzing a document image 1553 in accordance with an embodiment of theinvention. In the embodiment, pixels on document images (e.g., document1553) are labeled as one of four types. Pixels are labeled according tothe following association:

-   -   Type I: 32 by 32 pixel neighboring window (with the pixel under        consideration as the center) contains only maze pattern cells    -   Type II: 32 by 32 pixel neighboring window contains 60 or more        maze pattern cells    -   Type III: 32 by 32 pixel neighboring window contains 36 to 60        maze pattern cells    -   Type IV: 32 by 32 pixel neighboring window contains 35 or less        maze pattern cells        In the embodiment, the x-y position of a captured image can be        determined if the center of a captured image is located in type        I or type II areas and may be determined if the center is        located in a type III area. In FIG. 21, area 2101 corresponds to        a type I pixel area, area 2103 corresponds to a type II pixel        area, area 2107 corresponds to a type III pixel area, and area        2105 corresponds to a type IV pixel area. (The entire document        is analyzed and labeled but only a portion is shown in FIG. 21.)

In the embodiment, process 1500 may obtain digital document 1553 byrendering an electronic document to a bitmap or by scanning a paperdocument and then re-sampling the corresponding bitmap to an appropriateresolution. Determination of the resolution is based on the followingconsiderations: (1) the resolution of the document image shouldn't beless than that of the captured image, because the watermarked gray-scaledocument image will be matched with a camera-captured image in order todetermine the location of the captured image; (2) one printed mazepattern cell should be mapped to an integer number of document imagepixels so that the matching algorithm can work more effectively. Forexample, if the resolution of the camera is 0.15 mm/pixel, i.e., 0.15 mmin the physical world is mapped to one camera pixel, the printed mazepattern cell size is 0.45 mm*0.45 mm, i.e., a printed maze pattern cellis mapped to 3*3 pixels on the camera sensor, the resolution of thedocument image should also be set to 0.15 mm/pixel so that a printedmaze pattern cell will be mapped to a 3*3 pixel area in the documentimage.

Global Localization by Fast Image Match

If the x-y positions of the captured images of a stroke fail to bedetermined from m-array decoding, it is predictable that the centers ofall images are located at the area where x-y position cannot be uniquelydetermined. In this case, the captured images are warped by affinetransform obtained by maze pattern analysis, and then are matched withthe area of document image where x-y position cannot be uniquelydetermined. Once one frame is matched successfully, the locallocalization algorithm is used to locate the whole stroke.

With document 1553 (as shown in FIG. 15), the number of visible mazepattern cells in a 32 by 32 pixel sub-window typically varies from 10 to100. Pixels in the document are labeled with one of four types bywatermarked document analysis (step 1507 as shown in FIG. 15). A searchregion is set as the collection of type III and type IV areas.

FIG. 22 shows a result of a global localization for an exemplary stroke2201 in accordance with an embodiment of the invention. The x-y positionof the point 2203 on stroke 2201 is determined by global localization(corresponding to step 1509). Portion 2202 of a document is magnified tomagnified portion 2204 to better show point 2203 on stroke. (In theembodiment, only one point on stroke 2201 is determined by globallocalization, and the remaining points are determined by locallocalization.)

With global localization, a frame is matched with the document image atselect points of the search region. The original frame may be warped(transformed) by the scale and rotation (affine transform) from mazepattern analysis (corresponding to step 1503 as shown in FIG. 15). Anoffset may be useful for determining select points in matching a frame.

In the embodiment, the success of matching a frame corresponds to thelargest cross-correlation value between the captured image and thedocument image. (The cross-correlation value varies between 0 to 1.) Thelargest cross-correlation value should be sufficiently larger than othercross-correlation values and the threshold, i.e., how much larger, maybe determined by off-line training. For example, global localization mayrequire that the difference between the highest cross-correlation valueand the second highest cross-correlation value be larger than 0.1.

Local Localization by Fast Image Match

For those frames which x-y positions are not decoded/determined bym-array decoding/global localization, the locations should be near thelocations of the frames where x-y positions are determined.Consequently, the corresponding locations are determined by matching theimages with the neighbor areas of already determined x-y positions. Theperspective transform between captured images and document images isalso obtained.

FIG. 23 shows a result of a local localization for an exemplary stroke2301 in accordance with an embodiment of the invention. The locationpositions of a subset of points (e.g., points 2305 a and 2305 b) onstroke 2301 are determined by m-array decoding and the positions of theremaining points (e.g., points 2303 a and 2303 b) are determined bylocal localization.

In the embodiment, local localization utilizes start points and pivotalstart points. Start points are the location of frames which aresuccessfully located by m-array decoding (corresponding to step 1505 asshown in FIG. 15) or by global localization (corresponding to step 1509as shown in FIG. 15). A frame sequence is segmented into segments inwhich a local search is performed segment by segment. A segment may besplit into several segments during a local localization. In theembodiment, a local search is restricted to be within a search regionthat is based upon a pen tip's motion restrictions of velocity andacceleration.

The following procedure depicts an embodiment for local localization:

Frame Sequence Segmentation

(a) Select Pivotal Start Points

The first and the last start point in a stroke are pivotal start points.

For the start points between two pivotal start points p₁, p₂, the onewith the maximum distance D_(max) to the line L that passes through p₁,p₂, and D_(max) that is larger than a threshold (typically set at 0.5),is a pivotal start point.

(b) Refine the Perspective Transform at Pivotal Start Points

Get more accurate perspective transform by comparing the camera-capturedimage with the document image.

(c) Segment the Frame Sequence by Pivotal Start Points

Every segment starts from a pivotal start point or the first point ofthe stroke and ends at a pivotal start point or the last point of thestroke.

Determine Finished Segment

A segment is declared a finished segment when:

(a) There is at least one start point in this segment which is not apivotal start point, i.e., no more pivotal start points can be found forthis segment or in other words, the segment is very much like a straightline. In this case, all the points are interpolated and the segment isdeclared finished. or

(b) For every point in the segment, the corresponding frame has beenprocessed.

Find a Search Region for a Frame of Unfinished Segment

(a) The first point of the segment is a start point, and need not beprocessed.

(b) For the second point of the segment, the center of the search regionis set to be the first point, and the size of search region isrestricted by maximal velocity.

(c) For other points of the segment, the velocity at previous point thathas been processed can be estimated. The center of the search region canbe calculated from the location and velocity of the previous point, andthe size of search region is restricted by maximal acceleration.

Template Match in Search Region

This step is based on the assumption that there is only a little changeof a pen's pose in a short period of time, which means that there isonly a little difference of the perspective transform between adjacentframes.

(a) Warp the frame by the perspective matrix of previous processedframe.

(b) Determine the cross-correlation at every point of the search regionby matching the frame with the document image with the point as thecenter.

(c) The point with maximum cross-correlation should be the correctlocation of this frame.

Refine Perspective Transform for the Point

Refine the perspective transform by comparing the camera captured imagewith the document image.

Prune the Search Result

Several factors may cause error results to be obtained, such asincorrect initial perspective transform parameters and motion blur. Bythe motion restriction of velocity and acceleration, one can prune theerror results.

(a) All the start points should not be pruned.

(b) Go through the whole stroke from the first point to the last point.If a point cannot satisfy the motion restriction with the previouspoints, then this point must be wrong and should be pruned.

(c) Go though the whole stroke from the last point to the first point.If a point cannot satisfy the motion restriction with the previouspoints, then this point must be wrong and should be pruned.

(d) After the pruning, points kept in the stroke are correct points. Thepruned points are replaced by points determined from interpolation.

Camera-Pen-Tip Calibration

X-y positions obtained from m-array decoding and global/locallocalization represents the locations of the centers of captured images.To obtain the x-y positions of pen tip 202, the relationship between pentip 202 and the center of camera 203 needs to be calibrated. Fast andaccurate calibration is significant since ink cartridges may be changedfrequently.

By touching the pen-tip at a fixed contact point on the writing surfacein different postures, a few images are captured. The calibrationparameters are determined by using the constraint that all x-y positionsof these images should be mapped to the same point (the fixed contactpoint) by correct calibration parameters.

In the embodiment, the following procedure is used for calibrating a pentip. The procedure is used to estimate the calibration parameterL_(virtual-pentip):

-   -   a) Place the real pen tip at a fixed location L_(pentip) on        paper    -   b) Keep the pen tip at the location L_(pentip) standing, and        capture a series of images with different pen poses.    -   c) For each captured image, the transform F_(S→P) (which        transforms the position coordinates in the image captured by        camera to position coordinates in the real image on the paper)        and F_(P→S)=F_(S→P) ⁻¹ (the reverse of F_(S→P)) is calculated by        maze pattern analysis, m-array decoding and matching the        captured image with the document image, then:        L _(pentip) ^(i) =F _(S→P) ^(i) ·L _(virtual-pentip), i=1,2, . .        . , N        L _(virtual-pentip) ^(i) =F _(P→S) ^(i)·(L _(pentip) +ΔL _(i)),        i=1,2, . . . , N        where N is the number of captured images in the experiment, and        ΔP_(i) is the offset between the actual pen tip location in        i^(th) frame and L_(pentip).    -   d) Initialize {tilde over (L)}_(virtual-pentip) as (0, 0), where        L _(virtual-pentip) is the estimated value of        L_(virtual-pentip).    -   e) Using the first equation in (c), set L_(virtual-pentip) as L        _(virtual-pentip) to get L_(pentip) ^(i), i=1,2, . . . , N. By        averaging L_(pentip) ^(i), L_(pentip) is estimated as:

${\overset{\_}{L}}_{pentip} = {\frac{1}{N}{\sum\limits_{i = 1}^{i = N}\; L_{pentip}^{i}}}$

-   -   f) Using the second equation in (c), set L_(pentip) as L        _(pentip) to get L_(virtual-pentip) ^(i), i=1,2, . . . , N. By        averaging L_(virtual-pentip) ^(i), L_(virtual-pentip) is        estimated as:

${\overset{\_}{L}}_{{virtual} - {pentip}} = {\frac{1}{N}{\sum\limits_{i = 1}^{i = N}\; L_{{virtual} - {pentip}}^{i}}}$

-   -   g) Repeat step e. After several iterations, L_(virtual-pentip)        and L_(pentip) will converge respectively to substantially more        accurate results, which may be referred to as L_(virtual-pentip)        ^(accurate) and L_(pentip) ^(accurte).

Finally, one obtains L_(virtual-pentip) ^(accurate) as the calibrationparameter L_(virtual-pentip).

Map Images' Center Position to Pen Tip

X-y positions of pen tip 202 are determined by mapping the x-y positionsof image centers using the perspective transform obtained from locallocalization and calibration parameters. FIGS. 24 and 25 illustrate theresult of pen tip calibration of an example. FIG. 26 shows a recoveredstroke 2401 of pen tip 202 in accordance with an embodiment of theinvention. FIG. 25 shows an actual stroke 2501 that is associated withthe recovered stroke as shown in FIG. 24. Stroke 2403 in FIG. 24 showsthe path of the locations of the center of the captured images. That is,without calibration, stroke 2501 may be wrongly recovered (recovered asstroke 2403), whereas with proper calibration, stroke 2501 is correctlyrecovered (recovered as stroke 2401).

As can be appreciated by one skilled in the art, a computer system withan associated computer-readable medium containing instructions forcontrolling the computer system can be utilized to implement theexemplary embodiments that are disclosed herein. The computer system mayinclude at least one computer such as a microprocessor, digital signalprocessor, and associated peripheral electronic circuitry.

Although the invention has been defined using the appended claims, theseclaims are illustrative in that the invention is intended to include theelements and steps described herein in any combination or subcombination. Accordingly, there are any number of alternativecombinations for defining the invention, which incorporate one or moreelements from the specification, including the description, claims, anddrawings, in various combinations or sub combinations. It will beapparent to those skilled in the relevant technology, in light of thepresent specification, that alternate combinations of aspects of theinvention, either alone or in combination with one or more elements orsteps defined herein, may be utilized as modifications or alterations ofthe invention or as part of the invention. It may be intended that thewritten description of the invention contained herein covers all suchmodifications and alterations.

1. A method for determining a path of a pen tip within a document, themethod comprising: (A) decoding extracted bits associated with acaptured image to determine location coordinates of the captured image,by: (A)(i) obtaining the extracted bits that are associated with acaptured array; (A)(ii) processing the extracted bits to determinewhether the extracted bits contain at least one error bit and todetermine the location coordinates if no error bits are detected; and(A)(iii) if the at least one error bit is detected, further processingthe extracted bits to determine the location coordinates from a portionof the extracted bits, by: (A)(iii)(1) if an error bit is detected,selecting a different subset from the extracted bits, wherein at leastone bit of the different subset is not one of previously correctlydecoded bits; (A)(iii)(2) decoding associated bits of the differentsubset; (A)(iii)(3) in response to (2), determining whether anotherdecoding iteration shall be performed; (A)(iii)(4) if another decodingiteration shall be performed, selecting another subset from theextracted bits wherein at least one bit of the other subset is selectedfrom a set of wrongly decoded bits of each previous iteration andrepeating (2); and (A)(iii)(5) if another decoding iteration shall notbe performed, determining the location coordinates of the capturedarray, wherein the location coordinates are consistent with a localconstraint; (B) in response to (A), if the location coordinates of thecaptured image cannot be determined by decoding, matching the capturedimage with image information of the document; and (C) mapping the pathof the pen tip from the location coordinates of the captured image tothe location coordinates of the tip of the pen.
 2. The method of claim1, wherein the image information is selected from watermarked documentimage information or image information of a neighboring area of adecoded position corresponding to the captured image.
 3. The method ofclaim 1, further comprising: (D) analyzing a maze pattern to extractbits from the captured image, wherein the maze pattern corresponds to anm-array.
 4. The method of claim 3, further comprising: (B) segmentingthe maze pattern from an obstruction component of the captured image,wherein the obstruction component occludes the maze pattern.
 5. Themethod of claim 4, wherein (E) comprises: (E)(i) normalizing thecaptured image to compensate for non-uniform illumination.
 6. The methodof claim 1, wherein (B) comprises: (B)(i) matching the captured imagewith an area of an image of the document, wherein the area is estimatedby analyzing the image of the document or a corresponding location of aneighbor image frame, and wherein the corresponding location isdetermined by m-array decoding or global localization.
 7. The method ofclaim 6 wherein (B) further comprises: (B)(ii) in response to (i), ifthe location coordinates cannot be determined, warping the capturedimage; and (B)(iii) matching the captured image with an area of thedocument, wherein an x-y position cannot be uniquely determined bym-array decoding.
 8. The method of claim 7, wherein (B) furthercomprises: (B)(iv) in response to (iii), repeating (i).
 9. The method ofclaim 7, further comprising: (D) determining the area of the document,wherein the area is occluded by content of the document, and wherein thedocument is watermarked.
 10. The method of claim 1, further comprising:(D) calibrating the pen tip to obtain a calibration parameter, wherein(C) comprises using the calibration parameter and a perspectivetransform obtained from local localization.
 11. The method of claim 10,wherein (D) comprises: (D)(i) fixing the pen tip at a contact point onthe document; (D)(ii) changing a position of a pen camera center; and(D)(iii) mapping the position of the pen camera center to the contactpoint.
 12. The method of claim 10, wherein the calibration parameter isindicative of a virtual pen tip position.
 13. The method of claim 3,wherein (D) comprises: (D)(i) calculating a pattern parameter, thepattern parameter characterizing grid lines of the maze pattern.
 14. Themethod of claim 7, wherein (ii) comprises: (B)(ii)(1) scaling androtating a captured image by applying an affine transform obtained frommaze pattern analysis; and (B)(ii)(2) aligning the captured maze patternto a select point of a search region.
 15. The method of claim 9, wherein(D) comprises: (D)(i) obtaining a document image, wherein the documentimage is watermarked.
 16. A method for determining a path of a pen tipwithin a document, the method comprising: (A) decoding extracted bitsassociated with a captured image to determine location coordinates ofthe captured image; (B) in response to (A), if the location coordinatesof the captured image cannot be determined by decoding, matching thecaptured image with image information of the document, by: (B)(i)matching the captured image with an area of an image of the document,wherein the area is estimated by analyzing the image of the document ora corresponding location of a neighbor image frame, and wherein thecorresponding location is determined by m-array decoding or globallocalization; (B)(ii) in response to (i), if the location coordinatescannot be determined, warping the captured image, by: (B)(ii)( 1)scaling and rotating a captured image by applying an affine transformobtained from maze pattern analysis; (B)(ii)(2) aligning the capturedmaze pattern to a select point of a search region; and (B)(ii)(3)matching the warped frame with a document image, wherein the warpedframe corresponds to a largest cross-correlation value; and (B)(iii)matching the captured image with an area of the document, wherein an x-yposition cannot be uniquely determined by m-array decoding; and (C)mapping the path of the pen tip from the location coordinates of thecaptured image to the location coordinates of the tip of the pen. 17.The method of claim 16, wherein the image information is selected fromwatermarked document image information or image information of aneighboring area of a decoded position corresponding to the capturedimage.
 18. The method of claim 16, further comprising: (D) analyzing amaze pattern to extract bits from the captured image, wherein the mazepattern corresponds to an m-array.
 19. The method of claim 18, furthercomprising: (E) segmenting the maze pattern from an obstructioncomponent of the captured image, wherein the obstruction componentoccludes the maze pattern.
 20. The method of claim 19, wherein (B)comprises: (E)(i) normalizing the captured image to compensate fornon-uniform illumination.
 21. The method of claim 16, furthercomprising: (D) calibrating the pen tip to obtain a calibrationparameter, wherein (C) comprises using the calibration parameter and aperspective transform obtained from local localization.
 22. The methodof claim 21, wherein (D) comprises: (D)(i) fixing the pen tip at acontact point on the document; (D)(ii) changing a position of a pencamera center; and (D)(iii) mapping the position of the pen cameracenter to the contact point.
 23. The method of claim 21, wherein thecalibration parameter is indicative of a virtual pen tip position. 24.The method of claim 16, wherein (A) comprises: (A)(i) obtaining theextracted bits that are associated with a captured array; (A)(ii)processing the extracted bits to determine whether the extracted bitscontain at least one error bit and to determine the location coordinatesif no error bits are detected; and (A)(iii) if the at least one errorbit is detected, further processing the extracted bits to determine thelocation coordinates from a portion of the extracted bits, wherein thelocation coordinates are consistent with a local constraint.
 25. Themethod of claim 24, wherein (ii) comprises: (A)(ii)( 1) selecting afirst subset from the extracted bits; (A)(ii)(2) decoding the firstsubset; and (A)(ii)(3) in response to (A)(ii)(2), if no error bits aredetected, determining the location coordinates of the captured array.26. The method of claim 18, wherein (D) comprises: (D)(i) calculating apattern parameter, the pattern parameter characterizing grid lines ofthe maze pattern.
 27. The method of claim 16, wherein (C) comprises:(C)(i) calculating pen tip location coordinates from virtual pen tipcoordinates utilizing a perspective transform.
 28. A method fordetermining a path of a pen tip within a document, the methodcomprising: (A) decoding extracted bits associated with a captured imageto determine location coordinates of the captured image; (B) in responseto (A), if the location coordinates of the captured image cannot bedetermined by decoding, matching the captured image with imageinformation of the document, by: (B)(i) matching the captured image withan area of an image of the document, wherein the area is estimated byanalyzing the image of the document or a corresponding location of aneighbor image frame, and wherein the corresponding location isdetermined by m-array decoding or global localization; (B)(ii) inresponse to (i), if the location coordinates cannot be determined,warping the captured image; and (B)(iii) matching the captured imagewith an area of the document, wherein an x-y position cannot be uniquelydetermined by m-array decoding; (C) mapping the path of the pen tip fromthe location coordinates of the captured image to the locationcoordinates of the tip of the pen; and (D) determining the area of thedocument, wherein the area is occluded by content of the document, andwherein the document is watermarked, by: (D)(i) obtaining a documentimage, wherein the document image is watermarked; (D)(ii) determiningwhether a neighboring window of a pixel only contains maze patterncells, wherein the document image is represented by a plurality ofsub-windows; and (D)(iii) if the sub-window does not contain only themaze pattern, differentiating a measure of visible maze pattern cells.29. The method of claim 28, wherein (iii) comprises: (1) partitioningthe document image into a plurality of blocks having substantially asame size as maze pattern cells; (2) if corresponding small blocks areoccluded by document content, counting a number of completely visibleblocks in a neighboring window with a pixel as the center of the window;and (3) labeling the pixel with an indicator that is indicative of thenumber of visible blocks.
 30. The method of claim 28, wherein the imageinformation is selected from watermarked document image information orimage information of a neighboring area of a decoded positioncorresponding to the captured image.
 31. The method of claim 28, wherein(A) comprises: (A)(i) obtaining the extracted bits that are associatedwith a captured array; (A)(ii) processing the extracted bits todetermine whether the extracted bits contain at least one error bit andto determine the location coordinates if no error bits are detected; and(A)(iii) if the at least one error bit is detected, further processingthe extracted bits to determine the location coordinates from a portionof the extracted bits, wherein the location coordinates are consistentwith a local constraint.
 32. The method of claim 31, wherein (ii)comprises: (A)(ii)(1) selecting a first subset from the extracted bits;(A)(ii)(2) decoding the first subset; and (A)(ii)(3) in response to (2),if no error bits are detected, determining the location coordinates ofthe captured array.
 33. The method of claim 28, wherein (C) comprises:(C)(i) calculating pen tip location coordinates from virtual pen tipcoordinates utilizing a perspective transform.